Auto-scale vtgate with HPA #598

siadat · 2024-08-30T18:28:09Z

Problem

We'd like to be able to auto-scale number of vtgate replicas using HorizontalPodAutoscaler.

Solution

In order to make a CRD scalable by HPA, it should define the Scale subresource.

We also make sure if a vtgate is being autoscaled, the replicas value is not copied from VitessCluster. Otherwise, autoscaled values will be overwritten.

Context

https://vitess.slack.com/archives/CNE9WP677/p1725015568336609

pkg/controller/vitesscell/reconcile_vtgate.go

frouioui

Thanks for the contribution 🚀 Would it be possible to add an end-to-end test for this? We have examples of other features being covered by E2E tests under ./test/endtoend/.

pkg/operator/vtgate/horizontal_pod_autoscaler.go

frouioui · 2024-09-17T20:34:35Z

deploy/crds/planetscale.com_vitesscells.yaml

+                  autoscaler:
+                    properties:
+                      behavior:
+                        properties:


The ./test/endtoend/operator/operator-latest.yaml file we use for our tests must be updated with the newly compiled CRDs.

To do this, you can use the following command: kustomize build ./deploy > build/_output/operator.yaml. The output found in ./build/_output/operator.yamlcan be copied over the test file.

The changes made to the Deployment named vitess-operator will have to be reverted before pushing, as the new file you'll copy will override some defaults we use for our tests.

Finally, the changes made to ./test/endtoend/operator/operator-latest.yaml will have to be copied to the Vitess repository in a separate PR, into the file ./examples/operator/operator.yaml.

pkg/operator/vtgate/horizontal_pod_autoscaler.go

pkg/controller/vitesscell/reconcile_vtgate.go

frouioui

Looks good to me so far, outside of the few comments I left.

frouioui · 2024-09-25T16:54:17Z

test/endtoend/operator/cluster_autoscale.yaml

+    vtctld: vitess/lite:v20.0.1
+    vtgate: vitess/lite:v20.0.1
+    vttablet: vitess/lite:v20.0.1
+    vtorc: vitess/lite:v20.0.1
+    vtbackup: vitess/lite:v20.0.1


These tags should be latest. On main we want to test the latest tag of Vitess (main branch) with the PR's version of the vitess-operator - expect when running the upgrade test (which first uses the latest release of Vitess, and then upgrades to the latest tag of Vitess).

deploy/crds/planetscale.com_vitesscells.yaml

Makefile

siadat · 2024-10-04T01:12:26Z

Thanks for the reviews! I don't have access to see the buildkite logs to debug it.

Signed-off-by: Sina Siadat <[email protected]>

…tself Signed-off-by: Sina Siadat <[email protected]>

Signed-off-by: Sina Siadat <[email protected]>

It was using the name of the vtgate deployment, while it should be using the name of the VitessCell. Signed-off-by: Sina Siadat <[email protected]>

Signed-off-by: Sina Siadat <[email protected]>

Similar to 134463b Signed-off-by: Sina Siadat <[email protected]>

siadat · 2024-10-09T19:28:49Z

pkg/apis/planetscale/v2/vitesscell_types.go

+	// Behavior specifies the scaling behavior of the target in both Up and Down directions.
+	// +optional
+	Behavior *autoscalingv2.HorizontalPodAutoscalerBehavior `json:"behavior,omitempty"`
+}


@frouioui Let me know if I need to make any changes to this spec. It is similar to HorizontalPodAutoscaler, without the ScaleTargetRef field, whose value I am setting in https://github.com/siadat/vitess-operator/blob/sina/vitesscell-hpa/pkg/operator/vtgate/horizontal_pod_autoscaler.go#L46-L50

I think this is fine. If you don't want the end-user to customize ScaleTargetRef, which makes sense as it should be constant, then having our own AutoscalerSpec is okay.

frouioui

LGTM! Thanks for this valuable contribution.

Let's make sure the operator.yaml file is updated on the Vitess PR (vitessio/vitess#16805)

GuptaManan100

Great work! Thank you!

siadat force-pushed the sina/vitesscell-hpa branch from 990f4e0 to 718d930 Compare August 30, 2024 18:31

siadat commented Aug 30, 2024

View reviewed changes

pkg/controller/vitesscell/reconcile_vtgate.go Show resolved Hide resolved

siadat commented Aug 30, 2024

View reviewed changes

pkg/controller/vitesscell/reconcile_vtgate.go Outdated Show resolved Hide resolved

siadat force-pushed the sina/vitesscell-hpa branch from 718d930 to 11c8260 Compare August 30, 2024 18:50

siadat changed the title ~~Make VitessCell scalable by HPA~~ Scale vtgate with HPA Sep 10, 2024

siadat changed the title ~~Scale vtgate with HPA~~ Auto-scale vtgate with HPA Sep 10, 2024

siadat mentioned this pull request Sep 12, 2024

Autoscale vtgate replicas using HPA Yelp/paasta#3945

Closed

frouioui reviewed Sep 17, 2024

View reviewed changes

siadat mentioned this pull request Sep 18, 2024

Update example operator.yaml to support vtgate HPA vitessio/vitess#16805

Merged

5 tasks

siadat requested a review from frouioui September 18, 2024 23:30

mattlord self-requested a review September 23, 2024 11:56

frouioui reviewed Sep 25, 2024

View reviewed changes

siadat force-pushed the sina/vitesscell-hpa branch 3 times, most recently from be080bc to cc8b9c1 Compare September 30, 2024 18:16

lizard-boy mentioned this pull request Oct 1, 2024

Autoscale vtgate replicas using HPA grepdemos/paasta#7

Open

siadat force-pushed the sina/vitesscell-hpa branch 12 times, most recently from 4d7f153 to a2dc6ac Compare October 4, 2024 01:08

siadat force-pushed the sina/vitesscell-hpa branch from a2dc6ac to 18d0f0d Compare October 4, 2024 01:22

siadat added 16 commits October 8, 2024 23:07

Add HPA to watchResources in the vitesscell pkg

e044b59

Signed-off-by: Sina Siadat <[email protected]>

Update comments

5f6ca22

Signed-off-by: Sina Siadat <[email protected]>

Add endtoend test for HorizontalPodAutoscaler

e9a73b1

Signed-off-by: Sina Siadat <[email protected]>

Add hpa-test to Buildkite

3cbb0c2

Signed-off-by: Sina Siadat <[email protected]>

Use latest version of the components

a37d7c4

Signed-off-by: Sina Siadat <[email protected]>

Verify HPA before and after enabling autoscaler

e64ab2e

Signed-off-by: Sina Siadat <[email protected]>

Convert MaxReplicas from an optional to a mandatory value, like HPA i…

13319db

…tself Signed-off-by: Sina Siadat <[email protected]>

change order to match HPA

23decc9

Signed-off-by: Sina Siadat <[email protected]>

Update operator-latest.yaml after adding +listType=atomic

6b173f9

Signed-off-by: Sina Siadat <[email protected]>

Fix the value of VitessCell's .status.labelSelector

e081629

Signed-off-by: Sina Siadat <[email protected]>

Fix the HPA target reference name

a8bf2d2

It was using the name of the vtgate deployment, while it should be using the name of the VitessCell. Signed-off-by: Sina Siadat <[email protected]>

Make sure the VitessCluster reconciler does not overwrite replicas count

78d9ca1

Signed-off-by: Sina Siadat <[email protected]>

add metrics-server to provide metrics for HPA

1f34cfa

Signed-off-by: Sina Siadat <[email protected]>

verify hpa target and current utilization

1bfc3cc

Signed-off-by: Sina Siadat <[email protected]>

Update the name of HPA in integration test

58b817d

Signed-off-by: Sina Siadat <[email protected]>

Fix grep command in Buildkite step by avoiding the \1 back reference

3d41418

Signed-off-by: Sina Siadat <[email protected]>

siadat force-pushed the sina/vitesscell-hpa branch from 7886207 to 9fae65a Compare October 8, 2024 21:11

make generate

d885343

Signed-off-by: Sina Siadat <[email protected]>

siadat force-pushed the sina/vitesscell-hpa branch from 9fae65a to d885343 Compare October 8, 2024 21:15

siadat added 3 commits October 8, 2024 23:40

Update test/endtoend/operator/operator-latest.yaml

efe1409

Signed-off-by: Sina Siadat <[email protected]>

Update vtgate version, because we rebased

5392365

Signed-off-by: Sina Siadat <[email protected]>

update HPA verification grep expression

a843704

Signed-off-by: Sina Siadat <[email protected]>

siadat force-pushed the sina/vitesscell-hpa branch 2 times, most recently from 919c890 to 07efe52 Compare October 9, 2024 00:24

Port latest changes - change vt_dba grant

66ca9f2

Similar to 134463b Signed-off-by: Sina Siadat <[email protected]>

siadat force-pushed the sina/vitesscell-hpa branch from 07efe52 to 66ca9f2 Compare October 9, 2024 13:49

siadat commented Oct 9, 2024

View reviewed changes

frouioui approved these changes Oct 9, 2024

View reviewed changes

GuptaManan100 approved these changes Oct 14, 2024

View reviewed changes

frouioui merged commit 41a5e3f into planetscale:main Oct 14, 2024
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auto-scale vtgate with HPA #598

Auto-scale vtgate with HPA #598

siadat commented Aug 30, 2024 •

edited

Loading

frouioui left a comment

frouioui Sep 17, 2024

frouioui left a comment

frouioui Sep 25, 2024

siadat commented Oct 4, 2024 •

edited

Loading

siadat Oct 9, 2024

frouioui Oct 9, 2024

frouioui left a comment

GuptaManan100 left a comment

Auto-scale vtgate with HPA #598

Auto-scale vtgate with HPA #598

Conversation

siadat commented Aug 30, 2024 • edited Loading

Problem

Solution

Context

frouioui left a comment

Choose a reason for hiding this comment

frouioui Sep 17, 2024

Choose a reason for hiding this comment

frouioui left a comment

Choose a reason for hiding this comment

frouioui Sep 25, 2024

Choose a reason for hiding this comment

siadat commented Oct 4, 2024 • edited Loading

siadat Oct 9, 2024

Choose a reason for hiding this comment

frouioui Oct 9, 2024

Choose a reason for hiding this comment

frouioui left a comment

Choose a reason for hiding this comment

GuptaManan100 left a comment

Choose a reason for hiding this comment

siadat commented Aug 30, 2024 •

edited

Loading

siadat commented Oct 4, 2024 •

edited

Loading